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@ Method and apparatus for investigating and controlling an object 

(57) In order to investigate an object, such as a culture of micro-organisms, the Image of the object is 
repeatedly picked up at two different magnifications by a suitable image pick-up device (21 , 22, 24). The 
images at the two different magnifications are then analysed by a suitable picture image recognition 
device (25, 31) and the results of the repeated analysis are compared, thereby to derive a measurement 
of the change in the object with time. Thus, for a culture of micro-organisms, the number of cells can be 
determined at one magnification and the number of microscopic spherical bodies can be determined at 
another magnification, to obtain a measure of the activity of the culture. Preferably, one or more 
conditions of the object are then controlled on the basis of the measurement of change. 
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The present invention relates to a method and apparatus for investigating an object, and to a method and 
apparatus for controlling a condition of that object on the basis of the investigation. It is particularly, but not 
exclusively, concerned with the situation where that object Is a culture of micro-organisms. 

Culture systems based on living micro-organisms are widely used for 
5 (1 ) systems for the production of useful substances by culturing animal and/or plant cells, micro-organisms 

or yeast, for applications of those substances in the fields of medicine, diagnosis and foods, 

(2) systems of cell activation, e.g. for immunization therapy in the field of medical care, 

(3) systems for removing hazardous matter by using micro-organisms, for applications in the field of waste 
water treatment and (4) systems for farming of fish and shellfish in the field of fisheries. 

10 Research for further improving the practical use of such culture systems is proceeding, and it is desirable 
to improve the efficiency of all of these systems specifically, for the systems (1) and (4) improvement of pro- 
ductivity is desirable, for system (2) improvement of activation efficiency is desirable; and for system (3) 
improvement of operation stability is desirable. 

In system (1), the amount of useful substances produced by cells s extremely small, for example, a few 

15 ng/10 6 to a few ug/1 0 fl cells day. Therefore to obtain these useful substances on an industrial basis, improve- 
mentof cell concentration in the culture liquid, increase in the capacity of the culture tank and activation of cells 
(improvement of substance production ability of every cell) are desirable. To improve the cell concentration, 
by accelerating the multiplication of cells and suppressing death of the cells, it is necessary to collect detailed 
information about the concentration of the cells. For example, the ratio of the number of living cells (viability) 

20 to the total number of cells is desirable, as is the ratio of the number of cells having a potential for division (divi- 
sible cell ratio) to the number of living cells. For the activation of the cells, the ratio of the number of cells having 
a high secretion activity (secreting cell ratio) to the number of living cells is needed. 

Conventionally, these ratios were obtained by staining dead cells with a staining agent to distinguish the 
dead cells from living cells and thereafter a magnified image was observed with a microscope as disclosed in 

25 JP-A-2-27977 (1990). 

Currently, the majority of activity diagnosis methods which make use of the number of cells as an index 
are performed by human eyes. However, a few known methods make use of picture image processing and count 
cultured cells. 

JP-A-62-201332 (1987) and JP-A-64-029765 (1989) disclosed a method In which living cells (unstained) 

30 and dead cells (stained) were recognized separately by making use of a linear spatial filter. Also, JP-A-2-27977 
(1990) referred to above disclosed a method in which living cells and dead cells were discriminated by their 
sizes by making use of a linear spatial filter without injecting a staining agent into the culture liquid for the micro- 
organisms. However, both these known methods only obtained viability information. In a system for producing 
useful substances, Information concerning the ratio of secreting cells is particularly important Also, in system 

35 (2) mentioned above, information concerning the ratio of divisible cells, which represents an index indicating 
the ratio of active cells, is important. However no method of obtaining this Information under physiological con- 
ditions of cells has been proposed. 

In an activated sludge processing system (system (3), it is necessary to preventthe phenomenon of bulking, 
in which no active sludge is sedimented, in order to obtain stable operation, for the prevention thereof, micro- 

40 organism phase appearing in activated sludge groups is an important information and currently the sedimen- 
tation of activated sludge is evaluated on the basis of the ratio between aggregative micro-organisms and 
filamentous micro-organisms. 

JP-A-61 -21786 (1986) disclosed image picking-up with different magnifications when observing micro- 
organisms in sewage water. However no picture image processing was disclosed. 

45 JP-A-61 -32182 (1 988) disclosed a method in which images of celt specimens are obtained up with different 
magnifications and the cells were identified based upon the respective picture image information. In the method 
disclosed, only cells with no multiplication ability such as blood corpuscles, were considered, i.e. this method 
only considers objects which do not change substantially during the observation period. This document did des- 
cribe picture image processing of the picture images. 

so JP-A-64-53157 (1989) disclosed a method in which skin cell samples are observed by a microscope with 
one magnification. Then, those samples were observed by the microscope with a higher magnification after 
storing the current position and the characteristics of the cells were measured via picture image processing. 

As was mentioned above, the majority of systems currently used distinguished between living cells and 
dead celts, by staining the dead cells with a suitable staining agent. Subsequently, magnified images were 

55 observed by a human observer, to obtain a measurement of the viability of the sample. However, such a known 
system has a number of problems. 

(1) In general, a living body culture is a pure culture so that the culture system is isolated from the outside 
and It is necessary to preventthe introduction of living bodies other than those of the bodies to be cultured. 
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When specimens are taken from a culture liquid and a staining agent is injected, the closed system of the 
sterile culture system has to be temporarily opened, causing contact with the outside of the system. There- 
fore various germs may be introduced so that continuation of satisfactory culturing may be disturbed. 

(2) Since the culture liquid to which the staining agent has been injected cannot be returned to the culture 
5 tank, some culture [[quid is lost every time when specimens are taken so that any increase in the frequency 

of measurement of cell activity diagnosis is limited. 

(3) When the cells are caused to multiply in a transparent culture flask, although images of the cells can 
be observed by placing the flask on a microscope while the flask contains the liquid, a sampling sequence 
has to be included for injection of the staining agent. 

10 (4) It is difficult to judge whether the stained cells are Irving cells or dead cells even when such judgment 

is performed by a human or by recognition via a picture image processing. 

(5) No information about the ratio of cells capable of dividing is obtained. 

(6) No information about the ratio of cells capable of secreting is obtained. 

(7) Accordingly, it is difficult to increase the efficiency of the system. 

15 These problems usually arise even when the living bodies are micro-organisms and yeasts (including 
micro-organisms and yeasts produced by recombinant technology) and occur independent of whether the 
activity diagnosis is performed by eye or by a picture image processing. The method of judging whether cells 
are alive or dead, depending on the size distribution of the cultured cells via a picture image processing as dis- 
closed in JP-A-2-27977 (1990), may at least partially ameliorate problems (1)-(4) but cannot solve problems 

20 (5H7). 

Therefore, the present invention proposes that the optical investigations of an object at two different mag- 
nifications are performed repeatedly, and then the variation of the investigated properties may be determined. 

Preferably, the data from each Investigation of the variation of the properties is stored for each repetition 
of the optical investigations at the two different magnifications, and then the data so stored, for a plurality of 
25 repetitions of the investigation, may be compared. 

In a further development, the result of the investigation of the properties of the object may be used to control 
the object, by generating a control signal on the basis of that investigation of variations, which control signal 
controls means for determining at least one condition of the object 

With the present invention, within each repetition of the investigation at two different magnifications, the 
30 two measurements at the respectively different magnifications may be made simultaneously or sequentially. 

The present invention has a wide range of applications, but a preferred application is in the field of culturing 
of micro-organisms. As was discussed earlier, a culture of micro-organisms may contain cells corresponding 
to viable micro-organisms, and smaller particles which have been released from dead or dying micro-organ- 
isms. With the present invention, the concentration of cells in the culture of micro-organisms can be determined 
35 by counting at one magnification, and the concentration of the smaller particles can be obtained by counting 
the number thereof at a higher magnification. The ratio of these two concentrations gives a good measure of 
the viability of the culture of the micro-organisms. Thus, the culture of micro-organisms may be scanned by an 
image pick-up device, with that scanning preferably occurring in two mutually perpendicular directions. In this 
way, images of the micro-organisms in the culture may be obtained at two different magnifications. 
40 By obtaining such a viability measurement, it is then possible to control the conditions of the culture of micro- 

organisms so as to maintain the state of the culture substantially constant, at e.g. a high level of micro-organ- 
isms of the type desired. For example, when the culture is still growing, it is desirable to maximise the number 
of micro-organisms which are capable of dividing (and so increasing the numberof micro-organisms), and when 
a steady state is reached, it is desirable to maximise the numberof micro-organisms capable of secreting the 
45 substance which the culture is intended to produce. In general, it Is desirable to keep the ratio of small particles 
to cells to a value less than 2. 

Such control may be achieved in any suitable way, e.g. by controlling the supply and removal of culture 
material from a culture tank. 

The present invention is not limited to the culturing of micro- organisms however. It is also possible to make 
so use of the present system in a sewage plant, in which the micro-organisms in the active sewage are investi- 
gated. For example, the present invention may investigate the concentration of micro-organisms at one mag- 
nitude, and Investigate the shape of selected micro-organisms at another magnification. 

Afurtherapplication of the present invention is in the control of afurnace, In which the brightness distribution 
within the flame Is investigated at one magnification and then, at a higher magnification, a part of the flame is 
55 investigated to determine optically its temperature. 

The present invention relates to both methods and apparatus involving the aspects of the present invention 
discussed above. 

Embodiments of the invention will now be described in detail, by way of example, with reference to the 
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accompanying drawings, in which: 

Fig. 1 is a schematic diagram showing an example of an image pick-up device used in the present invention; 
Fig. 2 is a graph showing the relationship existing between the viability and the ratio of the numberof small 
spherical bodies to the numberof cells; 
5 Figs. 3(a) and 3(b) are schematic diagrams showing examples of cell shapes; 

Fig. 4 is a block diagram showing an example of a picture image recognition control system according to 
one embodiment of the present invention; 

Fig. 5 is a block diagram showing an example of a ceil culturing system according to a second embodiment 
of the present invention; 

10 Fig. 6 is a block diagram showing an example of a cell culturing system of a third embodiment of the present 

invention; 

Fig. 7 is a block diagram showing an example of a cell culturing system of a fourth embodiment of the pre- 
sent invention; Fig. 8 is a schematic diagram showing an example of a device for forming the two image- 
pick-up devices shown in Fig. 7; 
15 Fig. 9 Is a flowchart showing an example of a processing method, making use of a cell activity diagnosis 
method, according to a fifth embodiment of the present invention; 
Fig. 10(a) illustrates the brightness in the vicinity of the cell in an original picture image; 
Fig. 10(b) is a graph showing the relationship between pixel position and brightness corresponding to 
brightness conditions; 

20 Fig. 1 1 is a flowchart showing a sequence for living body recognition by means of a recognition method 
making use of brightness differences; 

Fig. 12 is a flowchart showing an example of discrimination between cells and microscopic spherical bodies 
which are identified In the processing method of Fig. 9, based upon the areas of the bodies; 
Fig. 13 is a flow chart showing an example of discrimination between ordinary cells and cells in the course 
25 of emitting microscopic spherical bodies, which cells are identified in the processing method of Fig. 9, based 

upon a shape coefficient; 

Fig. 14 is an example of processing when images of a culture liquid containing celts in suspension are iden- 
tified at a plurality of different magnifications; 

Fig. 15 is a flowchart showing an example of processing when a step, which estimates the variation of the 
30 current active state by exchange of data with means storing an earlier activity state, is included after the 
method shown in the flowchart of Fig. 9; 

Fig. 16 to Fig. 18 are respectively flowcharts showing examples of processing when a processing step, 
which controls factors affecting the state of culture liquid based upon the obtained activity data, is added 
after the methods shown in the flowcharts of Fig. 9, Fig. 14 and Fig. 15, respectively. 
35 Fig. 1 9 is a block diagram showing an example of a cell culturing system of a sixth embodiment of the pre- 
sent invention; 

Fig. 20 is a block diagram showing an example of a ceil culturing system of a seventh embodiment of the 
present invention; 

Fig. 21 is a schematic diagram of a practical cell culture system In which the present invention is embodied; 
40 Fig. 22 is a schematic diagram of a flame investigation system according to the present Invention; 

Fig. 23 is a schematic diagram of a sewage plant investigation system according to the present invention. 

First an example of the present invention will be discussed, considering a culture living bodies such as cells 
of animals and/or plants, yeasts and micro-organisms. These living bodies multiply via division or budding. The 
culture process generally moves from a logarithmic multiplication period in which the living bodies increase 
45 exponentially to a dying period in which the concentration thereof decreases via a multiplication period and a 
steady state period in which the apparent concentration thereof is substantially constant In culture systems, 
the aim is to achieve a condition of high density, by increasing the cell concentration in a short time and there- 
after maintaining the steady state period for a long time. 

The shapes of the living bodies, specifically the cells of animals and/or plants in the above culture p races- 
so ses, vary. In the logarithmic multiplication period and the multiplication period, almost all the cells are complete 
spherical bodies. However, when the culture is in the steady state period the spherical bodies distort and in 
the latter half of the steady state period, cells form polyp shaped projections on their surfaces. Furthermore, 
small spherical bodies are formed by projections which break off the cells. The relationship between these 
shapes and the activities of the cells is shown in Table 1. 

55 



5 



EP0 468 705 A2 



Table 1 
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25 The size of the cells of animals and/or plants is, in general, ten and a few um to a few tens of a um, and 
the size of the above small spherical bodies is of an order of a few urn. 

Fig. 2 illustrates experimental data showing the relationship between the ratio of the number of small spheri- 
cal bodies to the total number of cells and the viability of the culture. It can be seen that the two have a linear 
relationship. Hence, when the ratio of small spherical bodies to cells is determined, the viability of the culture 
so can be obtained. Furthermore, by Identifying the different shapes and counting the number of respective 
shapes, the ratio of secreting cells can be obtained. 

The identification of differences in shapes Is performed as follows. When the shape of the cell is a circle 
as shown in Fig. 3(a), the length of the circumference is expressed as follows, 

radius x 2 x it 

35 However, when there are recesses and protrusion on the circumference of the cell, as shown in Fig. 3(b), 
the length of the circumference gradually increases even when the area of two bodies are equal to each other. 
A value (shape coefficient) obtained by dividin g the c alculated circumferential length L based upon the radius 
obtained by the area of the circle, in that, r * Vareato, with the actual circumferential length L' as follows, 

2Vareaftc 

40 actual circumferential length 

This can be used as an Index indicating how the closeness of the shape of the body to a complete circle 
(when the body has more recesses and protrusions the value approaches 0 and when the shape is closer to 
a circle the value approaches 1). 

In binary picture images recognizing cells, the more active is the cell, (in other words, the higher the division 
45 activity), the more round is its shape. Accordingly, by making use of such a shape coefficient, the differences 
in shapes may be recognized and an activity diagnosis can be performed. 

For such activity diagnosis, the following characteristics are normally required. 

(a) Measurement of the total number of cells 

(b) Measurement of the number of small spherical bodies 
so (c) Calculation of the shape coefficient 

In such an arrangement, the microscopic spherical bodies are highly unlikely to be recognized by processing 
picture images obtained at a magnification which is appropriate for recognizing the cells. Since picture images 
of cells in suspension liquid, on many occasions, include stains as their background, a noise removing proces- 
sing is essential. However, the microscopic spherical bodies are likely ignored as noises, if such noise removing 
55 processing is used. 

On the other hand, with a magnification which is appropriate to identify microscopic spherical bodies, a 
single cell may be identified several times in a plurality of picture images. Therefore, when the number of cells 
are measured by counting, the errors in the counting become significant. 
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When the cell activity Is obtained by use of the shape coefficient, the recesses and protrusions around the 
circumference have to be recognized accurately. Therefore when a picture Image processing device is used 
in which the resolution (the number of pixels in one screen) is fixed, the magnification of the optical microscope 
has to be set relatively high. 

5 Thus, according to the present invention the processes (a) and (b) or (c) above have to be performed by 

changing the magnification of the optical investigation of the bodies. 

When the living bodies are yeast, they multiply by budding. Therefore, the relationship between their 
shapes and their activity is shown in Table 2. 

10 
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When there are many cells with protrusions, their multiplication activity is high and when there are many 
30 cells of spherical or elliptical shape, their secretion activity is high. This tendency Is thus opposite to that of the 
cells of animals and/or plants explained above. However, differences in shapes are recognized by the shape 
coefficient. Thus, the divisible cell rate and the secreting cell rate can be obtained. Again, the present invention 
proposes that the above processes (a) and (c) are performed by changing the magnification of the optical inves- 
tigation of the bodies. 

35 For activated sludge, when there are many aggregative bacteria, sedimentation is desirable and when there 

are many filamentous micro-organisms, sedimentation is undesirable. However the sedimentation varies 
depending upon the types of filamentous micro-organisms and floe shapes which are formed by the aggregative 
bacteria and the filamentous micro-organisms. Therefore it is necessary to obtain accurate information about 
the shapes. For this purpose, it is necessary to observe the ratio between the number of aggregative bacteria 

40 and the number of filamentous micro-organisms using a iow magnification, and to investigate the types of 
filamentous micro-organisms and the floe shapes using a high magnification. 

The total number of living bodies is determined by processing the picture images obtained using a low mag- 
nification, and the number and type (for filamentous micro-organism) of specific living bodies (such as small 
spherical bodies, cells with high division activity and cells with high secretion activity) are determined by pro- 

45 cessing picture images obtained using a high magnification. Based upon these data, information concerning 
the viability ratio of divisible cells and ratio of secretion cells is obtained to diagnose whether the culture envi- 
ronment is suitable. Based upon the diagnosis result, the culture environment may then be controlled to pre- 
determined conditions, to increase the efficiency of the system. 

The conditions of the culture environment which may have to be controlled include factors such as tem- 

50 perature, pH, osmotic pressure, dissolved oxygen concentration, retention time of the liquid, and living body 
concentration. Among these, temperature, pH, osmotic pressure and dissolved oxygen concentration are re- 
spectively controlled in suitable ranges for maintaining a high viability. For controlling the ratio of divisible cells 
and the ratio of secreting cells the value of (living body concentration) x (retention time of the liquid) Is adjusted. 
When the value of (living body concentration) x (retention time of the liquid) is increased, the ratio of secreting 

55 cells can be increased, and when the value (living body concentration) x (retention time of the liquid) is reduced, 
the ratio of divisible cells can be increased. 

In a useful substance production system, the value (living body concentration) x (retention time of the liquid) 
is set to be low at the Initial stage of culture so that a high density of the living bodies can be obtained in a short 
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time. Thereafter, the value of (living body concentration) x (retention time of the liquid) is set to be high to 
increase the secretion activity of the living bodies and to enhance the productivity of the substance. 

In a cell activating system, such as in immunization therapy, the value of (living body concentration) x 
(retention time of the liquid) is set to bB as small as possible to increase the concentration of active living bodies 

s so that it is as high as possible. 

In an activated sludge process, the value of (living body concentration) x (retention time of the liquid) is 
set to be large to decrease the ratio of divisible cells, to suppress any unnecessary increase in the concentration 
of the living bodies and to decrease excessive sludge, and as well to suppress multiplication of the filamentous 
micro-organism, to decrease the ratio of the filamentous micro-organism and to increase sedimentation. 

10 Embodiments of the present invention will now be explained with reference to the drawings. 

Fig. 4 shows an example of a picture image recognition control system according to one embodiment of 
the present invention. A tank 10 contains liquid containing living bodies in suspension for which reaction and 
multiplication be controlled. A part of the liquid is withdrawn from the tank 10 and passed to an image pick-up 
device 20. This image pick-up device, optically magnifies the image of the liquid at two optically different mag- 

15 nifi cations and the respective magnified images are picked-up as electrical picture image signals by an image 
picking-up means such as a television camera. 

These picture image signals are input to a picture image processing device 30 and the concentration of 
living bodies in the liquid is determined by processing the picture images obtained with a first magnification 
and the concentration of a specific type of living body is determined by processing the picture images obtained 

20 with a second magnification respectively. 

The resulting concentration data for the living bodies measured by the picture image processing device 30 
are input to a control device 40, from which the state of the liquid in the tank 1 0 may be controlled. 

Fig. 1 shows in more detail an example of the Image pick-up device 20. Atransparent container 21 is formed 
of a transparent material such as glass or plastics, and the object liquid for observation Is fed from the liquid 

25 tank 10 to the container. In this example, it is preferable to use the technology disclosed in JP-A-2-27977(1990) 
so as to allow handling of the observation object while maintaining the physiological conditions of the obser- 
vation object such as micro-organisms. 

An optical microscope 22 magnifies the image of the liquid in the transparent container 21. A television 
camera 24 converts the magnified image into electrical picture image signals (analogue signals). An AID con- 

30 verier 25 converts the analogue picture image signals from the television camera into a form of digital signals 
which can be processed In the picture image processing device 30, which are thereafter Input to the picture 
image processing device 30. An automatic objective lens exchange mechanism 23 changes the objective lens 
used by the optical microscope 22 in response to drive signals from the picture Image processing device and 
varies the magnification of the image of the liquid. 

35 Fig. 5 shows another embodiment of the present invention. In this embodiment components corresponding 
to components of the embodiment of Fig. 4 are indicated by the same reference numerals. In the embodiment 
of Fig. 5 when the drive signal issues from a timer 33, the liquid containing cells in suspension is sampled and 
fed to the image pick-up device 20. In this embodiment, the image pick-up device 20 has to have incorporated 
mechanism therein a such as a pump for supplying the liquid. The image pick-up device 20 magnifies the image 

40 of the liquid a plurality of times, with different magnifications, and photographs thereof are generated. The drive 
signal from the timer is simultaneously transmitted to a picture image recognition device 31 and the photo- 
graphed images of the liquid are processed without delay and the bodies may then be recognized. 

This picture image recognition device 31 generates (recognizes) digital picture images from a plurality of 
picture images fed to the image pick-up device 20. In such images, cells obtained by processing the picture 

45 image obtained at one magnification, and either one of microscopic spherical bodies emitted from cells, as cells 
emitting microscopic spherical bodies or both, which are obtained by processing the picture images obtained 
at a different magnification, all take the logical value "1". 

A picture image processing device 32 analyzes the binary picture image sent from the picture image rec- 
ognition device 31, measures the number of living bodies (either of cells, microscopic spherical bodies, cells 

so emitting microscopic spherical bodies) within the screen and calculates their respective concentrations in the 
liquid. A cell activity diagnosis device 50 calculates the ratio between the two variables, being the concentration 
of cells and the concentration of microscopic spherical bodies, this ratio, between two variables, being the con- 
centration of ceils and the concentration of cells emitting microscopic spherical bodies, or the ratio between 
three variables, being the concentration of cells, the concentration of microscopic spherical bodies and the con- 

55 centration of cells emitting microscopic spherical bodies. To obtain these ratios, use is made of the concen- 
tration of the living bodies obtained by the picture Image processing device 32. Hence, it is possible to diagnose 
the cell activity in the culture liquid. A culture condition control device 41 controls conditions affecting the state 
of liquid in a cell culture tank 1 0 based upon the results of the diagnosis carried out by the cell activity diagnosis 
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device 50. 

In the above examples, the drive signals from the timer 33 are transmitted only to the image pick-up device 
20 and the picture image recognition device 31. This is because the devices which receive information from 
the picture image processing device 32 are kept in a waiting condition for picture image signals so that 
5 synchronization of processing can be achieved. Of course, it is possible to arrange for ali of the devices to be 
driven by the timer. 

Fig. 6 shows a further embodiment of the present invention. In this embodiment, the picture images of the 
culture liquid photographed by an image pick-up device 30 are one type and a picture image recognition device 
31 recognizes the cells and the microscopic spherical bodies. Thereafter, a statistical value processing device 

w 34 discriminates celts from microscopic spherical bodies based upon the recognized areas of the living bodies, 
the numbers thereof and calculates the concentrations in the liquid. The functions of the cell activity diagnosis 
device 50 and the culture condition control device 41 are the same as in the embodiment shown in Fig. 5. 

In the above embodiments, the images are picked-up after the liquid is sampled. However another system 
can be used in which the image pick-up device is located directly in the culture tank and the images of the liquid 

15 are picked-up directly from within the tank 10. 

Fig. 7 shows a further embodiment of the present invention. The culture liquid sampled from a cell culturing 
tank is fed to two image pick-up devices 26 and 27. The magnification of the image pick-up device 26 is set at 
m and the device 26 generates picture images for observing cells. The magnification of the image pick-up device 
27 is set at n and the device 27 image generates picture images for observing particles other than cells. The 

20 picture images picked-up by the two image pick-up devices 26, 27 are input to a common picture image rec- 
ognition device 31 with a time displacement therebetween. The device 31 Identifies cells from the picture 
images from the image pick-up device 26 and particles from the picture images from the image pick-up device 
27. A statistical value processing device 34 statistically processes the number of cells and particles identified 
by the picture Image recognition device 31 and determines respective concentrations. In this embodiment, the 

25 functions of the cell activity diagnosis device 50 and the culture condition control device 41 are the same as in 
the embodiment shown in Fig. 5. 

Fig. 8 shows an example of a device for forming two image pick-up devices. The culture liquid sampled 
from a cell culturing tank 10 is split into two branches via a tube leading to the image pick-up devices and is 
fed to two transparent containers. The image is magnified by two optical microscopes 22 by setting the mag- 

30 nrfi cation of their objective lenses at m and n respectively and is picked-up by a television camera 24. The res- 
pective picture Image signals are converted into digital signals via two AID converters. Thus, use of totally iden- 
tical two sets of devices, in a hardware sense, is satisfactory. 

As shown in Fig. 2, there is a linear relationship between the number of the microscopic particles emitted 
into the culture liquid and the viability of the cells. Hence an activity determination system can be provided by 

35 making use of such relationship. It is necessary to determine parameters for that determination, however a suit- 
able determination means can be prepared based upon prior data. Furthermore, it is also possible to have 
means fordetermining the activity by comparing indexes obtained from the judging means prepared by the prior 
data and from the picked-up images of the present culture liquid. 

In another development of the present invention, a system performing measurement of concentrations of 

40 cells and microscopic spherical bodies can be formed which is provided with a cell activity diagnosis device in 
which the development of the activity of the cells is predicted by interpolating the speeds of Change of these 
concentrations at a future time. This interpolating calculation can be achieved by using differential equations. 

Another embodiment of the present invention Is illustrated in Fig. 19. In this embodiment culture liquid is 
sampled from a cell culturing tank 10 and fed to an image pick-up device 20. The magnified picture images of 

45 the culture liquid Image picked-up at an image pick-up device 20 are input to a picture image recognition device 
31 which identifies cells and microscopic spherical bodies or cells in the course of emitting microscopic spheri- 
cal bodies or both. A statistical processing device 34 discriminates the cells from the microscopic spherical 
bodies, determines the member thereof and determines the concentrations of both. A ceil activity diagnosis 
device 50 calculates the activity of the cells in the culture liquid based upon the concentrations of these bodies 

so and generates as output to e.g. a display and printer. Unlike the embodiment shown in Fig. 6, automatic control 
of the cell culturing tank 1 0 does not occur. The control depends on the operator who determines the diagnosis 
results. 

Another embodiment may be used which comprises a cell activity diagnosis device, which diagnosing cell 
activity in the culture liquid by comparing the current activity with prior data in a cell activity judging means. 
55 Another embodiment will now be discussed with reference to Fig. 20. A timer 33 generates driving signals 

which drive a pump of an image pick-up device 20. A culture liquid is sampled from a ceil culture tank 10 and 
fed to an image pick-up device 20. Picture images are picked-up a plurality of times with different magnifications 
and input to a picture image recognition device 31. The cells are recognized from picture images taken at one 
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magnification and the microscopic spherical bodies or the cells in the course of emitting microscopic spherical 
bodies or both are recognized from picture images taken at another magnification. A picture image processing 
device 32 analyzes the living bodies (cells, microscopic spherical bodies and cells in the course of emitting 
microscopic spherical bodies) recognized by the picture image recognition device 31 and determines the res- 

5 pective concentrations. By using the ratios calculated from cells and either one of two kinds of other living bodies 
or the both as indexes, a cell activity diagnosis device 50 diagnoses the activity of cells in the cell culturing 
tank 10. The difference between this embodiment and that of Fig. 5 is that in this embodiment the culture con- 
dition is not controlled automatically but instead the information about cell activity Is passed to the operator. 
Fig. 9 shows another embodiment using a cell activity diagnosis method. In step 110, images of culture 

10 liquid in which cells and microscopic spherical bodies are mixed in suspension are magnified and their images 
are picked-up. In step 120, bodies being either microscopic spherical bodies or cells in the course of emitting 
microscopic spherical bodies or both in the picture images picked-up at the step 110 are recognized by making 
use of brightness patterns around the living bodies in the picture images. In step 130, the concentrations of 
the cells and microscopic spherical bodies recognized in step 120 are measured for each kind of Irving bodies. 

15 In step 140, by using ratios calculated from the concentration of cells, the concentration of either microscopic 
spherical bodies or cells in the course of emitting microscopic spherical bodies or both, the activity of the cells 
in the culture liquid may be diagnosed. 

Fig. 10(a) and 10(b) illustrate the brightness pattern shown at the boundary of one kind of cell in a picture 
image. The part corresponding to the outline shows a low brightness while the part corresponding the cell shows 

20 a high brightness. Between the outline and the cell there exists a large brightness difference. Such a feature 
also occurs for the microscopic spherical bodies. Therefore, both cells and microscopic spherical bodies can 
be recognized via the same sequence. 

Fig. 1 1 shows one example of a method corresponding to the step 120 in Fig. 9, making use of the bright- 
ness patterns discussed above to recognize cells and microscopic spherical bodies. 

25 This recognition method is divided into a first stage in which a part of the living body Is recognised and a 

second stage in which the recognised part is extended, to obtain the entire image of the celt. 

In the step of recognizing a part of the cell, a minimum value filter is applied to the original picture image, 
and the original picture image is subtracted from that result (steps 210-220). These operations have, for two 
pixels located adjacent each other which have a large brightness difference therebetween, the effect of provi- 
so ding a brightness value of '1' for a pixel having a high brightness and '0' for a pixel having a low brightness 
value. When this operation is applied, to the picture images generated by picking up the image of the culture 
liquid of animal cells, the pixels corresponding to the outlines of the cells and small particles with small areas 
have a low brightness. Thus, picture images of variable brightness are obtained in which the parts Immediately 
inside the outlines show a high brightness. This picture image is fixed and binarized, and the parts with high 

35 brightness are selected. In this way, binarized picture images are obtained in which the number of pixels adja- 
cent the outline are recognized from the pixel groups corresponding to the insides of cells (step 230). 

There sometimes exists a large brightness difference between the pixels corresponding to the outline of 
the cells and those corresponding to the background of its outer circumference similar to the brightness differ- 
ence between the cell and its outline, such that processing In the steps 210-230 are likely erroneously to rec- 

40 ognize these differences as living bodies. However the brightness of these parts are normally low in comparison 
with the brightness level within the cells, so that these parts may be eliminated, with high reliability, by forming 
the logical product between the binarized picture images formed in step 230 and the binarized picture images 
obtained by directly binarizlng the original picture images (step 240-250). 

When the step of identifying a part of the living bodies has been completed, the areas of the respective 

45 living bodies which have been identified are small and the number of living bodies which have been identified 
is not exact because many parts of one Irving body may be identified. Therefore the identified part needs to be 
extended. This step makes use of the fact that the brightness of the inside of each cell is substantially constant. 

Therefore, the living body portion identified in the previous step is masked onto the original picture images 
(step 260). The term 'masking' means that the brightness values of the pixels in variable density picture images 

so having the same coordinates as pixels having a value of "1" in the binarized picture images are maintained 
unchanged and the brightness values of all the other pixels are made zero, to obtain variable brightness picture 
images. This processing operation causes the expansion starting point to be restricted to the part recognized 
in the prior step. 

Subsequently, to this variable brightness picture image a maximum value filter is applied (step 270). This 
55 process generates, variable brightness picture images in which the brightness of the pixels corresponding to 
the parts recognized in the earlier steps and the parts adjacent thereto takes positive values and the brightness 
values of the other pixels, being background pixels, take the value '0'. When the original picture image is sub- 
tracted from this variable brightness picture image, a variable brightness picture image is obtained in which 
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the background has a negative brightness, the outline has a high brightness and the inside of the body has a 
brightness about '0' (step 280). When this variable brightness picture image is binarized such that parts portions 
having a brightness between two threshold valves, for example between -5 and +15, are set to have a value 
"1", the parts corresponding to the insides of the living bodies are identified (step 290). The identified parts 

5 includes pixels which are located adjacent the living body prior to the expansion and have a small brightness 
difference from that part of the Irving body. Thus, expansion covering one pixel has been achieved. 

The logical product with the original picture image is also generated (step 300). The purpose of the 
generator of the logical product is to prevent expansion from the part having recognized the living body toward 
the background beyond the outline. 

10 The above recognition portion expansion step is repeated until no further expansion occurs. A standard 

number 1 of repetitions is 5 to 10. However even if the number of repetition is increased, there is no adverse 
effect on the identification of living bodies. Furthermore, the searching illustrated in Fig. 10 may be repeated 
in two mutually perpendicular directions A and B to increase accuracy. Fig. 10(b) then corresponds to direction 
A. 

15 Fig. 12 shows an example of discrimination processing of cells and microscopic spherical bodies, both of 
which were identified In step 120 of Fig. 9 based upon the identified areas of these living bodies. This process 
is included in a part of step 130 in Fig. 9. 

This discrimination method is carried out, after the area of each of the identified living bodies is measured, 
by determining whether the measured value is above a predetermined value. First, respective living bodies 

20 which are identified by the binary picture image are assigned respective numbers (labeling step 310). When 
the labeling is completed, the last number indicates the total number of living bodies. This value is substituted 
for the variable prepared for substituting the number of living bodies (step 320). Subsequently, in accordance 
with their number order discrimination is carried out, one by one, to determine whether the living body is a cell 
or a spherical body with a small area. Thus, the area of the living body with the corresponding number is 

25 measured (step 340). When the measured value is above a predetermined value, the living body is identified 
as a ceil and when the measured value is below the predetermined value the living body is identified as a spheri- 
cal body with a small area (step 350). This operation is repeated as many times as the number of living bodies 
(step 330). 

Fig. 13 shows a method in which discrimination between the normal cells and the cells in the course of 
30 emitting microscopic spherical bodies, which have been recognized together, is performed. This method makes 
use of a coefficient called the shape coefficient. The shape coefficient is a coefficient which indicates the degree 
of roundness of the shape of the living body and is determined by comparing the actual area of the living body 
with the area of the living body obtained from its circumference length when the living body is assumed to be 
a circle. This is expressed by the following equation. 

35 actual area 

(circumferential length/2) 2 x n 
The coefficient calculated by comparing the actual circumference of the living body with the circumference of 
the living body obtained from its area when the living body is a complete circle, as indicated below, may also 
be used. 

40 2Vareajot 

actual circumferential length 
When the call has many recesses and protrusions both values approach 0 and when the roundness of the cell 
is high, the values approach 1 . When the shape coefficient is used for cell observation, and when a microscopic 
spherical body is in the course of being divided from a cell, the coefficient is small. 

45 The method of Fig. 13 includes the following steps. First, respective living bodies which are identified in 
the binary picture image are assigned respective numbers (labeling step 31 0). When the labeling is completed, 
the last number indicates the total number of living bodies. This value is substituted for the variable prepared 
for substituting the number of living bodies (step 320). Subsequently, in accordance with their number order, 
discrimination is carried out, one by one, to determine whether the living body is a cell or a spherical body with 

so a small area. Thus, the area of the living body (step 340) and the circumferential length (step 360) of the living 
body are measured, and the shape coefficient thereof are calculated by making use of either of the above equ- 
ations for calculating the shape coefficient (step 370). Then, when the shape coefficient is above a predeter- 
mined value and near to 1, the body is identified as a normal cell and when the shape coefficient is below a 
predetermined value and near to 0 the body is identified as to be a cell in the course of emitting microscopic 

55 living body (step 380). These discrimination routines are repeated for each living body identified by numbers 
(step 330). 

Fig. 14 shows an example of the operation when the image of a culture liquid containing cells in suspension 
is image picked-up with a plurality of different magnifications. 
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In step 1 11, the image of the culture liquid containing cells in suspension is picked-up with a relatively low 
magnification. The cells existing in the picture image are recognized at a step 112. Then, in a step 112, the 
image of the culture liquid which has been sampled simultaneously in the step 1 1 1 is picked-up with a com- 
paratively high magnification, relative to that of the step 111. Either microscopic spherical bodies or cells in the 
5 course of emitting microscopic spherical bodies or the both may be identified from the picture image obtained 
in the step 112. In step 130, the concentrations of the cells and the microscopic spherical bodies are measured 
for every kind of living bodies, and in step 140 the activity of the culture liquid is diagnosed from the resultant 
concentrations. 

The present invention may provide a cell activity estimation and diagnosis method in which the cell activity 

w data when the culture liquid was previously sampled is stored. Then, in the data read out from the storage means 
and the current activity data may be substituted into an equation such as a differential equation. The variation 
of the activity of the cells may then be estimated. In Fig. 15, the steps 110 to 140 are the same as those in Fig. 
9. After these steps, a step 150 is carried out in which the variation of the activity of the cells is investigated by 
exchanging data with the storage means 160 which stores data representing previous activity. Another stage 

15 is then added subsequently, which controls factors affecting the culture liquid by making use of the resultant 
activity data. In Fig. 16 and Fig. 17, a step 170 is respectively shown, which controls factors affecting the state 
of the culture liquid by making use of the cell activity data obtained by the diagnosis result in step 140, after 
the same steps as in Fig. 9 and Fig. 14. Further, in Fig. 1 8, a step 1 70 is added, which controls factors affecting 
the state of the culture liquid by making use of both the activity diagnosis result from the step 140 and the esti- 

20 mation result in step 1 50, after the same steps as in Fig. 1 5. 

Another embodiment of the present invention is illustrated in Fig. 19. In mis embodiment, culture liquid is 
sampled from a cell culturing tank 10 and fed to an image pick-up device 20. The magnified picture images of 
the culture liquid image picked-up at an image pick-up device 20 are input to a picture image recognition device 
31 which Identifies cells and microscopic spherical bodies or cells in the course of emitting microscopic spheri- 

25 cal bodies or both. A statistical processing device 34 discriminates the cells from the microscopic spherical 
bodies, determines the member thereof and determines the concentrations of both. A cell activity diagnosis 
device 50 calculates the activity of the cells in the culture liquid based upon the concentrations of these bodies 
and generates as output to e.g. a display and printer. Unlike the embodiment shown in Fig. 6, automatic control 
of the cell culturing tank 10 does not occur. The control depends on the operator who determines the diagnosis 

30 results. 

Another embodiment may be used which comprises a cell activity diagnosis device, which diagnosing cell 
activity in the culture liquid by comparing the current activity with prior data in a celt activity judging means. 

Another embodiment will now be discussed with reference to Fig. 20. A timer 33 generates driving signals 
which drive a pump of an image pick-up device 20. A culture liquid is sampled from a cell culture tank 10 and 
35 fed to an image pick-up device 20. Picture images are pick up a plurality of times with different magnifications 
and input to a picture image recognition device 31. The cells are recognized from picture images taken at one 
magnification and the microscopic spherical bodies or the cells in the course of emitting microscopic spherical 
bodies or both are recognized from picture images taken at another magnification. A picture image processing 
device 32 analyzes the living bodies (cells, microscopic spherical bodies and cells in the course of emitting 
40 microscopic spherical bodies) recognized by the picture image recognition device 31 and determines the res- 
pective concentrations. By using the ratios calculated from cells and either one of two kinds of other living bodies 
or the both as indexes, a cell activity diagnosis device 50 diagnoses the activity of cells in the cell culturing 
tank 10. The difference between this embodiment and that of Fig. 5 is that in this embodiment the culture con- 
dition is not controlled automatically but instead the information about cell activity is passed to the operator. 
45 In the above embodiments, the type of cells is notspecified. However, a preferred application of the present 
invention is for animal cell culturing. Some animal cells have the characteristic that the cells emit microscopic 
spherical bodies, which cannot be said to be cells, before the cells are dead. Hence, the cell activity diagnosis 
method and the cell culturing method explained hereinabove can be applied easily. 

Figure 21 is a schematic diagram of a practical cell culture system in which the present invention is 
so embodied. Broth for the cell culture is mixed in a mixer 500 and passed via a filter 501 to a broth tank 502. 
Broth passes from the broth tank 502 via a temperature controller 503 to a duct 504, from which it may pass 
via a cell separator 505 to a culture vessel 506. A duct 507 containing a filter 508 may supply air, oxygen, carbon 
dioxide, and/or nitrogen to the culture vessel 506. By suitable operation of the flow of broth from the broth tank 
502 via the temperature control 503 and cell separator 505 to the culture vessel 506, a suitable culture of mi cro- 
ss organisms can be developed inside the culture vessel. At an appropriate time, a pump 507 is operated to with- 
draw broth from the culture vessel 506 via the cell separator 505 and the duct 504, and the separated broth is 
then passed to a collecting tank 508. 

In accordance with the present invention, an image pickup device 509 monitors the cells, microscopic small 
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particles, and celts in the course of emitting microscopic small particles, the image pick up device 509 operating 
in a manner corresponding to that described for the earlier embodiments. A controller 510 monitors the output 
of the image pick up device 509, to determine the viability of the culture in the cell vessel 506. The controller 
510 may then control the conditions of the culture in the culture vessel by one or more of: 
5 1 . supplying additional broth from the broth tank 502 at a suitable temperature determined by the tempera- 

ture controller 503; 

2. supplying one or more of air, oxygen, carbon dioxide and nitrogen, via the duct 507; 

3. removing broth from the culture vessel 506 by control of the pump 507, and passing that removed broth 
to the collecting tank 508. 

io In all the above embodiments, the present invention has been applied to the detection of a micro-organism 

in a culture thereof, but the present invention is not limited to the detection of micro-organisms. Figure 22 illus- 
trates an embodiment in which the flame of a furnace is monitored. In the embodiment of Fig. 22, a furnace 
600 has a nozzle 601 therein from which is generated a flame 602 for heating the furnace 600. An image pick 
up device 603 picks up the image of the flame 602 at two different magnifications, and passes respective signals 

15 to a signal processor 604 Including an analog-to-digital converter 605 and an output unit 606 which outputs 
digital signals representing the images photographed by image pick up device 603 to an image processor 607. 
The image processor 607 processes the images and passes them to a conversion analyzer 608. Display 609 
may show the image picked up by the image pick up device 603, whilst a character display 610 may display 
info±mation relating to the results of the analysis of the images by the conversion analyzer 608. Thus, the 

20 operator may be presented both with a visual indication of the image itself, on image display 609, and infab- 
matlon about the image on the character display 610. 

In this way, it is possible to investigate two different properties of the flame 602, by picking up the image 
of the flame 602 using the image pick up device 603 at two different magnifications. Thus, at a low magnification, 
in which substantially the whole of the flame can be seen, the variation in brightness across the flame 602 can 

25 be investigated. At higher magnifications, a more detailed investigation of the temperature of part of the flame 
602 can be obtained. A controller 611 may analyze the measurements carried out by the conversion analyzer 
608, and then may control ducts 612 and 613 respectively supplying fuel and air to the nozzle 601. In this way, 
the flame 602 can be controlled by the controller 61 1 independence on the observed image. Thus, the gener- 
ation of chemical species NOx and/or SOx may be observed and the fuel and air controlled accordingly. 

30 Figure 23 shows the application of the present invention to a sewage plant based on activated sludge. The 

sewage plant has a main tank 700 which receives sewage via an inlet pipe 701, the supply of sludge being 
controlled by a pump 702. The activity of the sewage 703 in the tank 700 may be controlled by the supply of 
oxygen via a duct 704, controlled by a pump 705, and treated sewage may be removed from the tank 703 via 
an outlet 706 controlled by a pump 707. 

35 In accordance with the present invention, an image pick up device 708 gets the image of the sewage 703 

at two different magnifications, to enable different properties to be investigated. For example, at one magnifi- 
cation, the total density of micro-organisms within the sewage 703 may be investigated, whilst at a higher mag- 
nification the density of a particular type of micro-organisms may be investigated. A controller 709 receives sig- 
nals from the image pick up device 708, and may then control the pumps 702, 705 and 707, accordingly to 

40 control the treatment of the sewage 703 in the tank 700. 

Thus, the present invention permits objects which change with time to be investigated at two different mag- 
nifications, to enable characteristics of the object to be determined. On thB basis of those characteristics, the 
conditions of the object may be varied, as desired. 

45 

Claims 

1. A method of controlling the state of an object (10), comprising: 

(i) optically investigating a first property of said object (10) at a first optical magnification, said optical 
so investigation of said first property including picking up the picture of said object (10) with an image pick- 
up device (20) to obtain a first image and processing said first image of said object (1 0), thereby to derive 
said first property; and 

(ii) optically investigating a second property of said object (10) at a second optical magnification, said 
optical investigation of said second property including picking-up said object (1 0) with an image pick-up 

55 device (20) to obtain a second image and processing said second image of said object (10), thereby 

to derive said second property; 
characterised in that 

said steps (i) and (ii) are repeated a plurality of times; the variation of said first and second properties 
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with time is investigated, thereby to derive a measurement of the change in said object (1 0) with time; and 
at least one condition of said object (1 0) is controlled on the basis of said variation of said first and 
second properties with time. 

2. A method according to claim 1 , wherein said controlling of said at least one condition is such as to coun- 
teract any change of said condition from a predetermined value. 

3. A method according to anyone of the preceding claims, wherein said optical investigating of said first prop- 
erty and said optical investigating of said second property occur simultaneously. 

4. A method according to any one of claims 1 or 2, wherein said optical investigating of said first property 
and said optical investigating of said second property occur sequentially. 

5. A method according to any one of the preceding claims, wherein said investigating of the variation of said 
first and second properties includes repeatedly storing data representing said properties for a plurality of 
investigations of said first and second properties, and subsequently comparing said data for said plurality 
of investigations. 

6. A method according to any one of the preceding claims, wherein said object is a sample of bodies of at 
least a first and a second type, and said first property is the concentration of said first type of body in said 
object, and said second property is the concentration of said second type of body in said object. 

7. A method according to claim 6, wherein said optical investigating of said first and second properties include 
scanning said sample of bodies in two substantially perpendicular directions. 

8. A method according to claim 6 or claim 7, wherein said first bodies are cells of living organisms and said 
second bodies are particles released from dying organisms. 

9. A method according to claim 8, wherein said investigating of said first and second properties includes deri- 
ving the ratio of said first and second properties. 

10. A method according to any one of claims 1 to 5, wherein said object is a flame, said first property is the 
brightness distribution of said object, and said second property is the temperature of at least a part of said 
object 

11. A method according to any one of claims 1 to 6, wherein said object is a sewage plant, said first property 
is the concentration of at least some of the micro-organisms in said sewage plant and said second property 
is the shape of selected micro-organisms in said sewage plant 

12. A method of culturing micro-organisms, comprising: 

a) establishing a culture of said micro-organisms; 

b) supplying culture material to said culture and removing culture material therefrom; 

c) optically investigating first bodies of said micro-organisms of said culture at a first optical magnifi- 
cation, said optical investigation of said first bodies including picking up the picture of said culture with 
an image pick-up device to obtain a first image and processing said first image, thereby to obtain the 
concentration of said first bodies in said culture; and 

d) optically investigating second bodies of said micro-organisms of said culture at a second optical mag- 
nification, said optical investigation of said second bodies Including picking up the image of said culture 
with an image pick-up device to obtain a second image and processing said second image, thereby to 
obtain the concentration of said second bodies in said culture; 

characterised in that 

said steps (c) and (d) are repeated a plurality of times, the variation of concentration of said first 
and second bodies is investigated and the supplying and removing of said culture material is controlled 
on the basis of said variation of concentration of said first and second bodies. 

13. A method according to claim 12, wherein said investigating of said variation of concentration includes deri- 
ving the ratio of said first and second bodies, and said controlling of said supplying and removing of said 
culture material is such as to maintain said ratio below a predetermined value of said ratio. 
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14. A method according to claim 14, wherein said first bodies are particles and said second bodies are cells 
and said predetermined value of said ratio is 2. 

15. A method of investigating a culture of micro-organisms, comprising: 
5 a) establishing said culture of said micro-organisms; 

b) optically investigating first bodies of said micro-organisms of said culture at a first optical magnifi- 
cation, said optical investigation of said first bodies including picking up the image of said culture with 
an image pick-up device to obtain a first image and processing said first image, thereby to obtain the 
concentration of said first bodies in said culture; 

10 c) optically investigating second bodies of said micro-organisms of said culture at a second optical mag- 

nification, said optical investigation of said second bodies including picking up the image of said culture 
with an image pick-up device to obtain a second image and processing said second image, thereby to 
obtain the concentration of said second bodies in said culture; 
characterised in that 

15 the method further includes: 

d) repeatedly storing data representing said concentration of said first and second bodies; and 

e) comparing said data for a plurality of investigations of said first and second properties. 

16. A method of culturing micro-organisms, comprising: 
20 a) establishing a culture of said micro-organisms; 

b) supplying culture material to said culture and removing culture material therefrom; 

c) optically investigating first bodies of said micro-organisms of said culture at a second optical magni- 
fication, said optical investigation of said first bodies including picking up the image of said culture with 
an image pick-up device to obtain a first image and processing said first image, thereby to obtain the 

is concentration of said first bodies in said culture; and 

d) optically Investigating second bodies of said micro-organisms of said culture at a second optical mag- 
nification, said optical investigation of said second bodies including picking up the image of said culture 
with an image pick-up device to obtain a second image and processing said second image, thereby to 
obtain the concentration of said second bodies in said culture; 

30 characterised in that 

said steps (c) and (d) are repeated a plurality of times, the variation of concentration of said first 
and second bodies is investigated and the supplying and removing of said culture material is controlled 
on the basis of said variation of concentration of said first and second bodies. 



35 17. A method of investigating a culture of micro-organisms, comprising: 

a) establishing said culture of said micro-organisms; 

b) optically investigating first bodies of said micro-organisms of said culture at a second optical mag- 
nification, said optical investigation of said first bodies including picking up the image of said culture 
with an image pick-up device to obtain a first Image and processing said first image, thereby to obtain 

40 the concentration of said first bodies in said culture; 

c) optically investigating second bodies of said micro-organisms of said culture at a second optical mag- 
nification, said optical investigation of said second bodies including picking up the image of said culture 
with an image pick-up device to obtain a second image and processing said second image, thereby to 
obtain the concentration of said second bodies in said culture; 

45 characterised in that 

the method further includes: 

d) repeatedly storing data representing said concentration of said first and second bodies; and 

e) comparing said data for a plurality of investigations of said first and second properties. 

so 18. An apparatus for controlling the state of an object, comprising: 

condition means for controlling at least one condition for said object (10); 
investigation means (20) for investigating said object optically, said investigation means including 
at least one image pick-up device (20) for picking up the image of said object at first and second magni- 
fications to derive at least first and second images; and 
55 processing means for processing said first and second images to derive first and second properties 

of said object 

characterised In that 

said investigation means (20) is arranged repeatedly to Investigate said object; and the apparatus 
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further includes: 

analysis means for comparing thefirstand second properties of said object derived bysaid repeated 
investigation by said investigating means, to derive an analysis result; and 

control means for automatically controlling said condition means on the basis of said analysis result. 

19. An apparatus according to claim 19, wherein said analysis means includes: 

means for storing data representing a plurality of investigations of said first and second properties 
of said object; and 

means for comparing said data representing said plurality of investigations. 

20. An apparatus for investigating the state of an object, comprising: 

investigation means for investigating said object optically, said investigation means including at 
least one image pick-up device for picking up the image of said object at first and second magnifications 
to derive at least first and second images; and 

processing means for processing said first and second images to derive first and second properties 
of said object; 

characterised in that 

said investigation means is arranged repeatedly to investigate said object; and the apparatus further 
includes: 

means for storing data representing a plurality of investigations of said first and second properties 
of said object; and 

means for comparing said data representing said plurality of investigations. 

21. An apparatus for controlling a culture of micro-organisms, comprising: 

container means for containing said culture of said micro-organisms; 

feed means for supplying and removing culture material form said container means; 

investigations means for investigating said culture optically, said investigation means including at 
least one image pick-up device for picking up said culture at first and second magnifications to derive first 
and second images; and 

processing means for processing said first and second images to derive concentrations of first and 
second bodies of said micro-organisms in said culture; 

characterised in that 

the investigation means is arranged repeatedly to investigate said object; and the apparatus further 
includes: 

analysis means for comparing the concentrations of said first and second bodies derived by said 
repeated investigation by said investigating means, to derive an analysis result; and 

control means for controlling said feed means on the basis of said analysis result 



16 



EP 0 468 705 A2 




Fig.1. 

17 



EP 0 468 705 A2 



When not stained 




Viability 

Fig.2. 



When stained 




Fig. 3a Fig. 3 b. 



« 



EP 0 468 705 A2 




EP 0 468 705 A2 




EP 0 468 705 A2 





21 



EP 0 468 705 A2 



Cells and microscopic spherical bodies mix, 
image of culture liquid in suspension is 
magnified and picked-up. 



Recognition of respective cells^microscopic 
spherical bodies and/or cells in course of 
emitting microscopic spherical bodies. 



Measurement of respective concentrations of 
cells, microscopic spherical bodies and / or 
cells m courseaf emitting microscopic 
spherical bodies. 



Activity diagnosis of culture liquid by 
making use of concentrations of cells 3 micro- 
scopic spherical bodies and/or cells in course 
of emitting microscopic spherical bodies. 



Fig. 9. 

22 



EP0 468 705 A2 



Fig. 10a. 



Position of y///^ 
cross-section f//f 


^^</y/\ Background 
ten \/ / / /-/ / /y 


f/s / 


Outline ^^Cry/// 




Pixel x 
*. - Outline 
#. . .Background 



23 



EP 0 468 705 A2 



(Original N 
image J 



11. 



Minimum value 
filter 



Subtraction 



^210 

^220 
'230 



240 



Binarization 



Binarization 



Logical product | v^/250 




Masking living body portion 
of binanzed picture image 
on original picture image 



Maximum value 
filter 



.260 



-270 



Subtraction 



-280 



Binarization 



-290 



Logical product 



-300 



Repeat 




(Picture irnage\ 
recognition J 



24 



EP0 468 705 A2 







.«/> >^<u 


>■» 




c u. - 


iving be 
; cell 








-JQ> 




25 



EP 0 468 705 A2 



LO ^ 8 




m 



11 



Jg 1 

— ai 
Q_ t_ 



ai 

.He 

v) )u 1/1 

2 2! ai 

£ o*-S 
o 



E 




en P 




,c E 




> X 




— a 














ai i/) 


ai 




e 


E 


ZD O 






e 




•S~'.> 



26 



i 



EP0 468 70SA2 





111 

) 


112 


Magnify image of culture 
t iquid with a tower 
magnification 




Magnify image of culture 
liquid with a higher 
magnification 






^122 


Recognition of cell 




Recognition of microscopic 
spherical bodies and/ or 
cells in course of emitting 
microscopic spherical bodies 









Measurement of respective 
concentrations of cells, microscopic 
spherical bodies and/or cells in course 
of emitting microscopic spherical 
bodies. 



Activity diagnosis of culture liquid by 
making use of concentrations of cells, 
microscopic spherical bodies and/ or 
cells in course of emitting microscopic 
spherical bodies. 




Fig.%. 



27 



EP 0 468 705 A2 



Cells and microscopic spherical 
bodies mix/image of culture 
liquid in suspension is 
magnified and picked-up 



Recognition of respective cells, 
microscopic spherical bodies 
and/or cells in course of emitting 
microscopic spherical bodies 



Measurement of respective 
concentrations of cells^micro- 
scopic spherical bodies and/ or 
cells in course of emitting micro - 
scopic spherical bodies 



Activity diagnosis of culture 
liquid by making use of 
concentrations of cells, micro- 
scopic spherical bodies and/or 
cells in course of emitting micro - / 
scopic spherical bodies 



Estimation of active state 
variation from now by putting 
together prior activity data 
and present data 



Fig.15. 



28 



EP 0 468 705 A2 



Cells and microscopic spherical 
bodies mix^ image of culture liquid 
in suspension is magnified and 
pickeo-up. 



Recognition of respective cells, 
microscopic spherical bodies and /or 
cells in course of emitting micro- 
scopic spherical bodies ✓ 



Measurement of respective 
concentrations of cells^micro- 
scopic spherical bodies and /or 
cells in courseof emitting micro - 
scopic spherical bodies. 



Activity diagnosis of culture liquid 
by making use of concentrations 
of cells, microscopic spherical 
bodies and/or cells in course of 
emitting microscopic spherical 
bodies. 



Control of factors affecting state 
of culture liquid by making use 
of cell activity data diagnosed in ^ 
step 140 



Fig. 16. 



29 



EP0 468 70S A2 





111 




112 

; 


Magnify image of culture 
liquid with a lower 
magnification 




Magnify image of culture 
liquid with a higher 
magnification 




,121 




^122 


Recognition of cell 




Recognition of microscopic 
spherical bodies and/ or 
cells in course of emitting 
microscopic spherical bodies 






i 



Measurement of respective 
concentrations of cells, microscopic 
spherical bodies and /or cells in course 
of emitting microscopic spherical 
bodies. 



,130 



Activity diagnosis of culture liquid by 
making use of concentrations of cells, 
microscopic spherical bodies and/or 
cells in course of emitting microscopic 
spherical bodies. 



140 



Control of factors affecting state 
of culture liquid by making use 
of cell activity data diagnosed 
in step 140 



Fig. 17. 



30 



EP 0 468 705 A2 



Cells and microscopic spherical 
bodies mix/image of culture 
liquid in suspension is 
magnified and picked-up 




Recognition of respective cells, 
microscopic spherical bodies 
and/or cells in course of emitting 
microscopic spherical bodies 



Measurement of respective 
concentrations of cells,micro- 
scopic spherical bodies and/ or 
cells in course of emitting micro - 
scopic spherical bodies 



Activity diagnosis of culture 
liquid by making use of 
concentrations of cells, micro- 
scopic spherical bodies and/or 
cells in course of emitting micro - 
scopic spherical bodies 







Estimation of active state , 
variation from now by putting 
together prior activity data 
and present data 






Control of factors affecting state 
of culture liquid by making use of 
cell activity data diagnosed in 
step 140 



.120 




Fig. 18. 



,140 




170 



31 



EP 0 468 705 A2 




32 



EP 0 468 705 A2 




33 



EP 0 468 705 A2 




34 



EP 0 468 705 A2 




Fig.22. 



35 



EP 0 468 705 A2 




